Load packages
Load the tidymodels library. This loads a collection of both tidymodels packages, and select tidyverse packages like dplyr, purrr, ggplot2. We will also load caret since it has some very nice datasets for regression and classification exercises.
library(tidymodels)
library(caret)
Loading required package: lattice
Registered S3 method overwritten by 'data.table':
method from
print.data.table
Attaching package: ‘caret’
The following objects are masked from ‘package:yardstick’:
precision, recall
The following object is masked from ‘package:purrr’:
lift
# Set random number seed to get consistent results
set.seed(101)
Regression example using Sacramento
Load dataset
data(Sacramento)
Sacramento
Sacramento
Data sampling
The first step in the modeling process is to split your data into separate training and testing datasets. The model will be trained using the training dataset, and the testing dataset will not be used until you are ready to assess model performance. The rsample::initial_split function helps with this initial splitting of the data. The rsample includes many other helpful functions for splitting the data for cross-validation, bootstrapping, etc.
# Split the dataset, using 75% of the data for training and 25% for testing
housing_split <- Sacramento %>%
as_tibble() %>%
dplyr::select(price,type,sqft,beds,baths,latitude,longitude) %>%
initial_split(prop = 0.75)
# This rsplit object tells you how many observations are used for training, how many for testing, and how many total
housing_split
<699/233/932>
# The training function can be used to extract the training data from the rsplit object
housing_training <- housing_split %>%
training()
housing_training
# The testing function can be used to extract the testing data from the rsplit object
housing_testing <- housing_split %>%
testing()
housing_testing
Data pre-processing
After splitting the data, we will do some data processing. To do this, we will use the recipe package. A recipe is a blueprint for how data will be processed. By creating a blueprint, rather than processing data directly, we can apply the same blueprint to training and testing datasets. Importantly, the recipe is defined using only data from the training dataset, which will allow us to see how well the model performs using the testing dataset. Recipe steps can be defined using pipes with a number of sequential steps - there are many many options for recipe steps.
Model training
Next, we will use the parsnip package to define a number of models. Generally, we use parsnip to define 3 things about our model:
- The type of model (e.g., linear regression or random forest)
- the mode of the model (e.g., regression or classification)
- The engine for the model (e.g.,
ranger or randomForest)
After we’ve defined the model in this way, we can use the fit function to fit the model.
# Define and fit a linear regression model
housing_model_lm <- linear_reg() %>%
set_engine("lm")
housing_model_lm
Linear Regression Model Specification (regression)
Computational engine: lm
housing_fit_lm <- housing_model_lm %>%
fit(price ~ ., data = housing_training_juiced)
housing_fit_lm
parsnip model object
Call:
stats::lm(formula = formula, data = data)
Coefficients:
(Intercept) sqft beds baths
248531 106863 -22235 6528
latitude longitude type_Multi_Family type_Residential
5953 17524 -4297 10493
# Define and fit a random forest regression model using the randomForest engine/package
housing_model_randomForest <- rand_forest(trees = 100, mode = "regression") %>%
set_engine("randomForest")
housing_model_randomForest
Random Forest Model Specification (regression)
Main Arguments:
trees = 100
Computational engine: randomForest
housing_fit_randomForest <- housing_model_randomForest %>%
fit(price ~ ., data = housing_training_juiced)
housing_fit_randomForest
parsnip model object
Call:
randomForest(x = as.data.frame(x), y = y, ntree = ~100)
Type of random forest: regression
Number of trees: 100
No. of variables tried at each split: 2
Mean of squared residuals: 6238597435
% Var explained: 64.71
# Define and fit a random forest regression model using the ranger engine/package
housing_model_ranger <- rand_forest(trees = 100, mode = "regression") %>%
set_engine("ranger")
housing_model_ranger
Random Forest Model Specification (regression)
Main Arguments:
trees = 100
Computational engine: ranger
housing_fit_ranger <- housing_model_ranger %>%
fit(price ~ ., data = housing_training_juiced)
housing_fit_ranger
parsnip model object
Ranger result
Call:
ranger::ranger(formula = formula, data = data, num.trees = ~100, num.threads = 1, verbose = FALSE, seed = sample.int(10^5, 1))
Type: Regression
Number of trees: 100
Sample size: 699
Number of independent variables: 7
Mtry: 2
Target node size: 5
Variable importance mode: none
Splitrule: variance
OOB prediction error (MSE): 6103318793
R squared (OOB): 0.655258
Once we have the model fits, we can use the predict function to generate our predictions for our testing dataset. The predict function always produces a dataframe with the same number of rows as observations. Because of this, bind_cols can be used to bind the predictions to the original dataframe
# Generate predictions for our testing using the ranger model
predict(housing_fit_ranger, housing_testing_baked)
# Add these ranger predictions to the testing dataset
housing_fit_ranger %>%
predict(housing_testing_baked) %>%
bind_cols(housing_testing_baked)
# Save this combined dataframe for later
housing_ranger_predict <- housing_fit_ranger %>%
predict(housing_testing_baked) %>%
bind_cols(housing_testing_baked) %>%
# Add a column for model name
mutate(model_name = "ranger")
# Let's do the same thing for the linerar regression model
housing_lm_predict <- housing_fit_lm %>%
predict(housing_testing_baked) %>%
bind_cols(housing_testing_baked) %>%
mutate(model_name = "lm")
# Let's do the same thing for the randomForest model
housing_randomForest_predict <- housing_fit_randomForest %>%
predict(housing_testing_baked) %>%
bind_cols(housing_testing_baked) %>%
mutate(model_name = "randomForest")
# Let's combine all of these datasets so we can look at them side-by-side
housing_all_predict <- bind_rows(housing_lm_predict,
housing_ranger_predict,
housing_randomForest_predict)
Let’s just look and see how our predictions line up with the observed values in our testing dataset.
housing_all_predict %>%
ggplot(aes(x = price,y=.pred,color=model_name)) +
geom_point() +
geom_smooth(method = "lm") +
labs(x = "Observed price",
y = "Predicted price",
title = "Predictions vs observed values for 3 model types\nA simple linear regression is overlaid") +
coord_equal()

Model performance assessment
Using our predictions from the parsnip package, we can use the yardstick package to generate model performance metrics. This can be done using the metrics function, which generates a default metric set (for a regression model, these are root mean squared error or rsme, r-squared or rsq, and mean absolute error or mae; for a classification model, these are accuracy and Kappa or kap). You can also define a custom set of metrics using metric_set, and there are also individual functions for all metric types.
housing_ranger_predict %>%
# Here we use the metric functions and must define the truth value and the estimated prediction
metrics(truth = price, estimate = .pred)
When the predictions are in a dataframe, we can group by model type and calculat metrics by group
housing_all_predict%>%
group_by(model_name) %>%
metrics(truth = price, estimate = .pred)
housing_all_predict%>%
group_by(model_name) %>%
metrics(truth = price, estimate = .pred)%>%
ggplot(aes(x = model_name, y = .estimate)) +
geom_bar(stat="identity") +
facet_wrap(.~.metric,scales="free") +
labs(x = "Model name",
y = "Model performance metric estimate",
title = "Model performance metrics for 3 model types")

We can also do what we just did in a much more tidy fashion, while also keeping the model specifications, model fits, model predictions, and model metrics all in a single dataframe. This ensures that things stay together, and makes it very easy to extract summary statistics or plots. purrr::map and list columns makes this all possible. We could apply this same approach to build and test many models for cross-validation, for hyperparameter tuning, etc.
# Define a tibble using model names and their associated specifications
set.seed(101)
all_models <-
tibble(model_name = "lm",
model = list(housing_model_lm)) %>%
add_row(model_name = "ranger",
model = list(housing_model_ranger)) %>%
add_row(model_name = "randomForest",
model = list(housing_model_randomForest))
all_models
all_model_results <- all_models %>%
# Add a column for model fits
mutate(model_fit = purrr::map(model,
~fit(.x, price ~ ., data = housing_training_juiced)),
# Add a column for predictions
model_predictions = purrr::map(model_fit,
~.x %>%
predict(housing_testing_baked) %>%
bind_cols(housing_testing_baked)),
# Add a column for model metrics
model_metrics = purrr::map(model_predictions,
~metrics(.x, truth = price, estimate = .pred)))
all_model_results
# This plot is the same as the one we made above
all_model_results %>%
unnest(model_metrics) %>%
ggplot(aes(x = model_name, y = .estimate)) +
geom_bar(stat="identity") +
facet_wrap(.~.metric,scales="free") +
labs(x = "Model name",
y = "Model performance metric estimate",
title = "Model performance metrics for 3 model types")

Classification example using GermanCredit
Let’s also go through a classification example using the GermanCredit dataset from caret. Now we will try to predict credit rating (good or bad) using a number of predictors.
Load packages
data(GermanCredit)
GermanCredit
Data pre-processing
# Split the credit dataset, using 75% of the data for training and 25% for testing, stratified by credit class
# This maintains the ratio of Good and Bad credit classes in both the training and testing datasets
credit_split <- GermanCredit %>%
as_tibble() %>%
# Convert most columns to factors since they are binaries
mutate_at(vars(-Duration,-Amount,-InstallmentRatePercentage,-ResidenceDuration,-Age,-NumberExistingCredits,-NumberPeopleMaintenance),
as.factor) %>%
initial_split(prop = 0.75, strata = "Class")
# The training function can be used to extract the training data from the rsplit object
credit_training <- credit_split %>%
training()
# The testing function can be used to extract the testing data from the rsplit object
credit_testing <- credit_split %>%
testing()
credit_recipe <- credit_training %>%
recipe(Class ~.) %>%
# Remove all near-zero variance predictors, such as factors with only one level
step_nzv(all_predictors()) %>%
# step_corr removes highly correlated variables
step_corr(all_numeric()) %>%
# step_center normalizes data to have a mean of 0
step_center(all_numeric()) %>%
# step_scale normalizes data to have a standard deviation of 0
step_scale(all_numeric())%>%
# Make all factors dummy columns
step_dummy(all_nominal(),-all_outcomes())
credit_recipe_prepped <- credit_recipe %>%
# prep trains the recipe using the training dataset
prep()
# Use use the juice function to apply the prepped recipe to the training dataset
credit_training_juiced <- juice(credit_recipe_prepped)
# We use the bake function to apply the prepped recipe to the testing dataset
credit_testing_baked <- credit_recipe_prepped %>%
bake(credit_testing)
Model training
# Define and fit a logistic regression model
credit_model_lr <- logistic_reg() %>%
set_engine("glm")
# Define and fit a random forest regression model using the randomForest engine/package
credit_model_randomForest <- rand_forest(trees = 100, mode = "classification") %>%
set_engine("randomForest")
# Define and fit a random forest regression model using the ranger engine/package
credit_model_ranger <- rand_forest(trees = 100, mode = "classification") %>%
set_engine("ranger")
# Define a tibble using model names and their associated specifications
all_models_credit <-
tibble(model_name = "ranger",
model = list(credit_model_ranger)) %>%
add_row(model_name = "randomForest",
model = list(credit_model_randomForest))
all_model_results_credit <- all_models_credit %>%
# Add a column for model fits
mutate(model_fit = purrr::map(model,
~fit(.x, Class ~ ., data = credit_training_juiced)),
# Add a column for class predictions
model_predictions_class = purrr::map(model_fit,
~.x %>%
predict(credit_testing_baked, type = "class") %>%
bind_cols(credit_testing_baked)),
# Add a column for model metrics from class predictions
model_metrics = purrr::map(model_predictions_class,
~metrics(.x, truth = Class, estimate = .pred_class)),
# Add a column for probability predictions
model_predictions_prob = purrr::map(model_fit,
~.x %>%
predict(credit_testing_baked, type="prob") %>%
bind_cols(credit_testing_baked)),
# Add ROC curves from probability predictions
roc_curves = purrr::map(model_predictions_prob,
~roc_curve(.x, Class, .pred_Good)))
LS0tCnRpdGxlOiAiVGlkeW1vZGVscyBsaXZlIHNlc3Npb24gLSBFY29EYXRhU2NpZW5jZSIKYXV0aG9yOiAiR2F2aW4gTWNEb25hbGQgLSBFbnZpcm9ubWVudGFsIE1hcmtldHMgU29sdXRpb25zIExhYiAoZW1MYWIpIgpkYXRlOiAiMTEvMTkvMjAxOSIKb3V0cHV0OiBodG1sX25vdGVib29rCi0tLQoKIyBJbnRyb2R1Y3Rpb24gCgpUaGlzIGxpdmUgc2Vzc2lvbiBoYXMgYmVlbiBhZGFwdGVkIGZyb20gZnJvbSBFZGdhciBSdWl6J3MgW0EgR2VudGxlIEludHJvZHVjdGlvbiB0byB0aWR5bW9kZWxzXShodHRwczovL3J2aWV3cy5yc3R1ZGlvLmNvbS8yMDE5LzA2LzE5L2EtZ2VudGxlLWludHJvLXRvLXRpZHltb2RlbHMvKS4gV2Ugd2lsbCBydW4gdGhyb3VnaCBhbiBlbmQtdG8tZW5kIG1vZGVsaW5nIGV4YW1wbGVzLiBUaGUgZmlyc3Qgd2lsbCBiZSBhIHJlZ3Jlc3Npb24gZXhhbXBsZSB1c2luZyB0aGUgYFNhY3JhbWVudG9gIGhvdXNpbmcgcHJpY2VzIGRhdGFzZXQsIGFuZCB0aGUgc2Vjb25kIHdpbGwgYmUgYSBjbGFzc2lmaWNhdGlvbiBleGFtcGxlIHVzaW5nIHRoZSBgR2VybWFuQ3JlZGl0YCBjcmVkaXQgc2NvcmUgZGF0YXNldC4gCgpJIHdpbGwgYnJpZWZseSB0b3VjaCBvbiBpbXBvcnRhbnQgc3RlcHMgb2YgdGhlIHByZWRpY3RpdmUgbW9kZWxpbmcgcHJvY2VzcywgYnV0IHRoaXMgaXMgKm5vdCogbWVhbiB0byBiZSBhIGNvbXByZWhlbnNpdmUgaW5zdHJ1Y3Rpb24gb24gcHJlZGljdGl2ZSBtb2RlbGluZyAoKmEuay5hLiogbWFjaGluZSBsZWFybmluZykuIFRoaXMgdHV0b3JpYWwgaXMgcmF0aGVyIGp1c3QgbWVhbnQgdG8gZ2l2ZSBhIGZsYXZvciBvZiB3aGF0J3MgcG9zc2libGUgdXNpbmcgdGhlIG5ldyBgdGlkeW1vZGVsc2AgdW5pdmVyc2Ugb2YgcGFja2FnZXMuIEZvciBhIG11Y2ggbW9yZSBkZXRhaWxlZCBvdmVydmlldywgc2VlIE1heCBLdWhuJ3MgZmFudGFzdGljIFtBcHBsaWVkIFByZWRpY3RpdmUgTW9kZWxpbmddKGh0dHA6Ly9hcHBsaWVkcHJlZGljdGl2ZW1vZGVsaW5nLmNvbSkuCgojIEluc3RhbGwgbmVjZXNzYXJ5IHBhY2FrZ2VzCgpgYGB7cn0KIyBwYWNtYW4gd2lsbCBoZWxwIHVzIGluc3RhbGwgYW55IG5lY2Vzc2FyeSBwYWNrYWdlcwppZiAoIXJlcXVpcmUoInBhY21hbiIpKSBpbnN0YWxsLnBhY2thZ2VzKCJwYWNtYW4iKQojIHBhY21hbjo6cF9sb2FkIGNoZWNrcyB0byBzZWUgaWYgdGhpcyBwYWNrYWdlcyBhcmUgaW5zdGFsbGVkLCBhbmQgaW5zdGFsbHMgdGhlbSBpZiBub3QKcGFjbWFuOjpwX2xvYWQodGlkeW1vZGVscywgcmFuZ2VyLCByYW5kb21Gb3Jlc3QsIGNhcmV0KQpgYGAKCgojIExvYWQgcGFja2FnZXMKCkxvYWQgdGhlIHRpZHltb2RlbHMgbGlicmFyeS4gVGhpcyBsb2FkcyBhIGNvbGxlY3Rpb24gb2YgYm90aCB0aWR5bW9kZWxzIHBhY2thZ2VzLCBhbmQgc2VsZWN0IHRpZHl2ZXJzZSBwYWNrYWdlcyBsaWtlIGRwbHlyLCBwdXJyciwgZ2dwbG90Mi4gV2Ugd2lsbCBhbHNvIGxvYWQgY2FyZXQgc2luY2UgaXQgaGFzIHNvbWUgdmVyeSBuaWNlIGRhdGFzZXRzIGZvciByZWdyZXNzaW9uIGFuZCBjbGFzc2lmaWNhdGlvbiBleGVyY2lzZXMuCgpgYGB7cn0KbGlicmFyeSh0aWR5bW9kZWxzKQpsaWJyYXJ5KGNhcmV0KQojIFNldCByYW5kb20gbnVtYmVyIHNlZWQgdG8gZ2V0IGNvbnNpc3RlbnQgcmVzdWx0cwpzZXQuc2VlZCgxMDEpCmBgYAoKIyBSZWdyZXNzaW9uIGV4YW1wbGUgdXNpbmcgYFNhY3JhbWVudG9gCgojIyBMb2FkIGRhdGFzZXQKCmBgYHtyfQpkYXRhKFNhY3JhbWVudG8pCgpTYWNyYW1lbnRvCmBgYAoKIyMgRGF0YSBzYW1wbGluZyAKClRoZSBmaXJzdCBzdGVwIGluIHRoZSBtb2RlbGluZyBwcm9jZXNzIGlzIHRvIHNwbGl0IHlvdXIgZGF0YSBpbnRvIHNlcGFyYXRlIHRyYWluaW5nIGFuZCB0ZXN0aW5nIGRhdGFzZXRzLiBUaGUgbW9kZWwgd2lsbCBiZSB0cmFpbmVkIHVzaW5nIHRoZSB0cmFpbmluZyBkYXRhc2V0LCBhbmQgdGhlIHRlc3RpbmcgZGF0YXNldCB3aWxsIG5vdCBiZSB1c2VkIHVudGlsIHlvdSBhcmUgcmVhZHkgdG8gYXNzZXNzIG1vZGVsIHBlcmZvcm1hbmNlLiBUaGUgYHJzYW1wbGU6OmluaXRpYWxfc3BsaXRgIGZ1bmN0aW9uIGhlbHBzIHdpdGggdGhpcyBpbml0aWFsIHNwbGl0dGluZyBvZiB0aGUgZGF0YS4gVGhlIGByc2FtcGxlYCBpbmNsdWRlcyBtYW55IG90aGVyIGhlbHBmdWwgZnVuY3Rpb25zIGZvciBzcGxpdHRpbmcgdGhlIGRhdGEgZm9yIGNyb3NzLXZhbGlkYXRpb24sIGJvb3RzdHJhcHBpbmcsIGV0Yy4KCmBgYHtyfQojIFNwbGl0IHRoZSBkYXRhc2V0LCB1c2luZyA3NSUgb2YgdGhlIGRhdGEgZm9yIHRyYWluaW5nIGFuZCAyNSUgZm9yIHRlc3RpbmcKaG91c2luZ19zcGxpdCA8LSBTYWNyYW1lbnRvICU+JQogIGFzX3RpYmJsZSgpICU+JQogIGRwbHlyOjpzZWxlY3QocHJpY2UsdHlwZSxzcWZ0LGJlZHMsYmF0aHMsbGF0aXR1ZGUsbG9uZ2l0dWRlKSAlPiUKICBpbml0aWFsX3NwbGl0KHByb3AgPSAwLjc1KQoKIyBUaGlzIHJzcGxpdCBvYmplY3QgdGVsbHMgeW91IGhvdyBtYW55IG9ic2VydmF0aW9ucyBhcmUgdXNlZCBmb3IgdHJhaW5pbmcsIGhvdyBtYW55IGZvciB0ZXN0aW5nLCBhbmQgaG93IG1hbnkgdG90YWwKaG91c2luZ19zcGxpdAoKIyBUaGUgdHJhaW5pbmcgZnVuY3Rpb24gY2FuIGJlIHVzZWQgdG8gZXh0cmFjdCB0aGUgdHJhaW5pbmcgZGF0YSBmcm9tIHRoZSByc3BsaXQgb2JqZWN0CmhvdXNpbmdfdHJhaW5pbmcgPC0gaG91c2luZ19zcGxpdCAlPiUKICB0cmFpbmluZygpCgpob3VzaW5nX3RyYWluaW5nCgojIFRoZSB0ZXN0aW5nIGZ1bmN0aW9uIGNhbiBiZSB1c2VkIHRvIGV4dHJhY3QgdGhlIHRlc3RpbmcgZGF0YSBmcm9tIHRoZSByc3BsaXQgb2JqZWN0CmhvdXNpbmdfdGVzdGluZyA8LSBob3VzaW5nX3NwbGl0ICU+JQogIHRlc3RpbmcoKQoKaG91c2luZ190ZXN0aW5nCmBgYAoKIyMgRGF0YSBwcmUtcHJvY2Vzc2luZyAKCkFmdGVyIHNwbGl0dGluZyB0aGUgZGF0YSwgd2Ugd2lsbCBkbyBzb21lIGRhdGEgcHJvY2Vzc2luZy4gVG8gZG8gdGhpcywgd2Ugd2lsbCB1c2UgdGhlIGByZWNpcGVgIHBhY2thZ2UuIEEgcmVjaXBlIGlzIGEgYmx1ZXByaW50IGZvciBob3cgZGF0YSB3aWxsIGJlIHByb2Nlc3NlZC4gQnkgY3JlYXRpbmcgYSBibHVlcHJpbnQsIHJhdGhlciB0aGFuIHByb2Nlc3NpbmcgZGF0YSBkaXJlY3RseSwgd2UgY2FuIGFwcGx5IHRoZSBzYW1lIGJsdWVwcmludCB0byB0cmFpbmluZyBhbmQgdGVzdGluZyBkYXRhc2V0cy4gSW1wb3J0YW50bHksIHRoZSByZWNpcGUgaXMgZGVmaW5lZCB1c2luZyBvbmx5IGRhdGEgZnJvbSB0aGUgdHJhaW5pbmcgZGF0YXNldCwgd2hpY2ggd2lsbCBhbGxvdyB1cyB0byBzZWUgaG93IHdlbGwgdGhlIG1vZGVsIHBlcmZvcm1zIHVzaW5nIHRoZSB0ZXN0aW5nIGRhdGFzZXQuIFJlY2lwZSBzdGVwcyBjYW4gYmUgZGVmaW5lZCB1c2luZyBwaXBlcyB3aXRoIGEgbnVtYmVyIG9mIHNlcXVlbnRpYWwgc3RlcHMgLSB0aGVyZSBhcmUgbWFueSBtYW55IG9wdGlvbnMgZm9yIHJlY2lwZSBzdGVwcy4KCmBgYHtyfQoKaG91c2luZ19yZWNpcGUgPC0gaG91c2luZ190cmFpbmluZyAlPiUKICAjIFNwZWNpZnkgcmVncmVzc2lvbiBtb2RlbCBmb3JtdWxhCiAgcmVjaXBlKHByaWNlIH4uKSAgJT4lCiAgIyBzdGVwX2NvcnIgcmVtb3ZlcyBoaWdobHkgY29ycmVsYXRlZCB2YXJpYWJsZXMKICBzdGVwX2NvcnIoYWxsX251bWVyaWMoKSkgJT4lCiAgIyBzdGVwX2NlbnRlciBub3JtYWxpemVzIGRhdGEgdG8gaGF2ZSBhIG1lYW4gb2YgMAogIHN0ZXBfY2VudGVyKGFsbF9udW1lcmljKCksIC1hbGxfb3V0Y29tZXMoKSkgJT4lCiAgIyBzdGVwX3NjYWxlIG5vcm1hbGl6ZXMgZGF0YSB0byBoYXZlIGEgc3RhbmRhcmQgZGV2aWF0aW9uIG9mIDAKICBzdGVwX3NjYWxlKGFsbF9udW1lcmljKCksIC1hbGxfb3V0Y29tZXMoKSkgJT4lCiAgIyBDcmVhdGUgZHVtbXkgdmFyaWFibGUgY29sdW1ucyBmb3IgYWxsIGZhY3RvciBjb2x1bW5zCiAgc3RlcF9kdW1teShhbGxfcHJlZGljdG9ycygpLC1hbGxfbnVtZXJpYygpKQoKaG91c2luZ19yZWNpcGUKCmhvdXNpbmdfcmVjaXBlX3ByZXBwZWQgPC0gaG91c2luZ19yZWNpcGUgJT4lCiAgIyBwcmVwIHRyYWlucyB0aGUgcmVjaXBlIHVzaW5nIHRoZSB0cmFpbmluZyBkYXRhc2V0CiAgcHJlcCgpCgpob3VzaW5nX3JlY2lwZV9wcmVwcGVkCgojIFVzZSB1c2UgdGhlIGp1aWNlIGZ1bmN0aW9uIHRvIGFwcGx5IHRoZSBwcmVwcGVkIHJlY2lwZSB0byB0aGUgdHJhaW5pbmcgZGF0YXNldApob3VzaW5nX3RyYWluaW5nX2p1aWNlZCA8LSBqdWljZShob3VzaW5nX3JlY2lwZV9wcmVwcGVkKQoKaG91c2luZ190cmFpbmluZ19qdWljZWQKCiMgV2UgdXNlIHRoZSBiYWtlIGZ1bmN0aW9uIHRvIGFwcGx5IHRoZSBwcmVwcGVkIHJlY2lwZSB0byB0aGUgdGVzdGluZyBkYXRhc2V0CmhvdXNpbmdfdGVzdGluZ19iYWtlZCA8LSBob3VzaW5nX3JlY2lwZV9wcmVwcGVkICU+JQogIGJha2UoaG91c2luZ190ZXN0aW5nKSAKCmhvdXNpbmdfdGVzdGluZ19iYWtlZApgYGAKCiMjIE1vZGVsIHRyYWluaW5nCgpOZXh0LCB3ZSB3aWxsIHVzZSB0aGUgYHBhcnNuaXBgIHBhY2thZ2UgdG8gZGVmaW5lIGEgbnVtYmVyIG9mIG1vZGVscy4gR2VuZXJhbGx5LCB3ZSB1c2UgcGFyc25pcCB0byBkZWZpbmUgMyB0aGluZ3MgYWJvdXQgb3VyIG1vZGVsOgoKMS4gVGhlIHR5cGUgb2YgbW9kZWwgKGUuZy4sIGxpbmVhciByZWdyZXNzaW9uIG9yIHJhbmRvbSBmb3Jlc3QpICAKMi4gdGhlIG1vZGUgb2YgdGhlIG1vZGVsIChlLmcuLCByZWdyZXNzaW9uIG9yIGNsYXNzaWZpY2F0aW9uKSAgCjMuIFRoZSBlbmdpbmUgZm9yIHRoZSBtb2RlbCAoZS5nLiwgYHJhbmdlcmAgb3IgYHJhbmRvbUZvcmVzdGApIAoKQWZ0ZXIgd2UndmUgZGVmaW5lZCB0aGUgbW9kZWwgaW4gdGhpcyB3YXksIHdlIGNhbiB1c2UgdGhlIGBmaXRgIGZ1bmN0aW9uIHRvIGZpdCB0aGUgbW9kZWwuICAKCmBgYHtyfQojIERlZmluZSBhbmQgZml0IGEgbGluZWFyIHJlZ3Jlc3Npb24gbW9kZWwKaG91c2luZ19tb2RlbF9sbSA8LSBsaW5lYXJfcmVnKCkgJT4lCiAgc2V0X2VuZ2luZSgibG0iKSAKCmhvdXNpbmdfbW9kZWxfbG0KCmhvdXNpbmdfZml0X2xtIDwtIGhvdXNpbmdfbW9kZWxfbG0gJT4lCiAgZml0KHByaWNlIH4gLiwgZGF0YSA9IGhvdXNpbmdfdHJhaW5pbmdfanVpY2VkKQoKaG91c2luZ19maXRfbG0KCiMgRGVmaW5lIGFuZCBmaXQgYSByYW5kb20gZm9yZXN0IHJlZ3Jlc3Npb24gbW9kZWwgdXNpbmcgdGhlIHJhbmRvbUZvcmVzdCBlbmdpbmUvcGFja2FnZQpob3VzaW5nX21vZGVsX3JhbmRvbUZvcmVzdCA8LSAgcmFuZF9mb3Jlc3QodHJlZXMgPSAxMDAsIG1vZGUgPSAicmVncmVzc2lvbiIpICU+JQogIHNldF9lbmdpbmUoInJhbmRvbUZvcmVzdCIpIAoKaG91c2luZ19tb2RlbF9yYW5kb21Gb3Jlc3QKCmhvdXNpbmdfZml0X3JhbmRvbUZvcmVzdCA8LSBob3VzaW5nX21vZGVsX3JhbmRvbUZvcmVzdCAlPiUKICBmaXQocHJpY2UgfiAuLCBkYXRhID0gaG91c2luZ190cmFpbmluZ19qdWljZWQpCgpob3VzaW5nX2ZpdF9yYW5kb21Gb3Jlc3QKCiMgRGVmaW5lIGFuZCBmaXQgYSByYW5kb20gZm9yZXN0IHJlZ3Jlc3Npb24gbW9kZWwgdXNpbmcgdGhlIHJhbmdlciBlbmdpbmUvcGFja2FnZQpob3VzaW5nX21vZGVsX3JhbmdlciA8LSByYW5kX2ZvcmVzdCh0cmVlcyA9IDEwMCwgbW9kZSA9ICJyZWdyZXNzaW9uIikgJT4lCiAgc2V0X2VuZ2luZSgicmFuZ2VyIikgCgpob3VzaW5nX21vZGVsX3JhbmdlcgoKaG91c2luZ19maXRfcmFuZ2VyIDwtIGhvdXNpbmdfbW9kZWxfcmFuZ2VyICU+JQogIGZpdChwcmljZSB+IC4sIGRhdGEgPSBob3VzaW5nX3RyYWluaW5nX2p1aWNlZCkKCmhvdXNpbmdfZml0X3JhbmdlcgpgYGAKCk9uY2Ugd2UgaGF2ZSB0aGUgbW9kZWwgZml0cywgd2UgY2FuIHVzZSB0aGUgYHByZWRpY3RgIGZ1bmN0aW9uIHRvIGdlbmVyYXRlIG91ciBwcmVkaWN0aW9ucyBmb3Igb3VyIHRlc3RpbmcgZGF0YXNldC4gVGhlIHByZWRpY3QgZnVuY3Rpb24gYWx3YXlzIHByb2R1Y2VzIGEgZGF0YWZyYW1lIHdpdGggdGhlIHNhbWUgbnVtYmVyIG9mIHJvd3MgYXMgb2JzZXJ2YXRpb25zLiBCZWNhdXNlIG9mIHRoaXMsIGBiaW5kX2NvbHNgIGNhbiBiZSB1c2VkIHRvIGJpbmQgdGhlIHByZWRpY3Rpb25zIHRvIHRoZSBvcmlnaW5hbCBkYXRhZnJhbWUKCmBgYHtyfQojIEdlbmVyYXRlIHByZWRpY3Rpb25zIGZvciBvdXIgdGVzdGluZyB1c2luZyB0aGUgcmFuZ2VyIG1vZGVsCnByZWRpY3QoaG91c2luZ19maXRfcmFuZ2VyLCBob3VzaW5nX3Rlc3RpbmdfYmFrZWQpCgojIEFkZCB0aGVzZSByYW5nZXIgcHJlZGljdGlvbnMgdG8gdGhlIHRlc3RpbmcgZGF0YXNldApob3VzaW5nX2ZpdF9yYW5nZXIgJT4lCiAgcHJlZGljdChob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogIGJpbmRfY29scyhob3VzaW5nX3Rlc3RpbmdfYmFrZWQpCgojIFNhdmUgdGhpcyBjb21iaW5lZCBkYXRhZnJhbWUgZm9yIGxhdGVyCmhvdXNpbmdfcmFuZ2VyX3ByZWRpY3QgPC0gaG91c2luZ19maXRfcmFuZ2VyICU+JQogIHByZWRpY3QoaG91c2luZ190ZXN0aW5nX2Jha2VkKSAlPiUKICBiaW5kX2NvbHMoaG91c2luZ190ZXN0aW5nX2Jha2VkKSAlPiUKICAjIEFkZCBhIGNvbHVtbiBmb3IgbW9kZWwgbmFtZQogIG11dGF0ZShtb2RlbF9uYW1lID0gInJhbmdlciIpCgojIExldCdzIGRvIHRoZSBzYW1lIHRoaW5nIGZvciB0aGUgbGluZXJhciByZWdyZXNzaW9uIG1vZGVsCmhvdXNpbmdfbG1fcHJlZGljdCA8LSBob3VzaW5nX2ZpdF9sbSAlPiUKICBwcmVkaWN0KGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgYmluZF9jb2xzKGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgbXV0YXRlKG1vZGVsX25hbWUgPSAibG0iKQoKIyBMZXQncyBkbyB0aGUgc2FtZSB0aGluZyBmb3IgdGhlIHJhbmRvbUZvcmVzdCBtb2RlbApob3VzaW5nX3JhbmRvbUZvcmVzdF9wcmVkaWN0IDwtIGhvdXNpbmdfZml0X3JhbmRvbUZvcmVzdCAlPiUKICBwcmVkaWN0KGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgYmluZF9jb2xzKGhvdXNpbmdfdGVzdGluZ19iYWtlZCkgJT4lCiAgbXV0YXRlKG1vZGVsX25hbWUgPSAicmFuZG9tRm9yZXN0IikKCiMgTGV0J3MgY29tYmluZSBhbGwgb2YgdGhlc2UgZGF0YXNldHMgc28gd2UgY2FuIGxvb2sgYXQgdGhlbSBzaWRlLWJ5LXNpZGUKaG91c2luZ19hbGxfcHJlZGljdCA8LSBiaW5kX3Jvd3MoaG91c2luZ19sbV9wcmVkaWN0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBob3VzaW5nX3Jhbmdlcl9wcmVkaWN0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICBob3VzaW5nX3JhbmRvbUZvcmVzdF9wcmVkaWN0KQpgYGAKCkxldCdzIGp1c3QgbG9vayBhbmQgc2VlIGhvdyBvdXIgcHJlZGljdGlvbnMgbGluZSB1cCB3aXRoIHRoZSBvYnNlcnZlZCB2YWx1ZXMgaW4gb3VyIHRlc3RpbmcgZGF0YXNldC4KCmBgYHtyfQpob3VzaW5nX2FsbF9wcmVkaWN0ICU+JQogIGdncGxvdChhZXMoeCA9IHByaWNlLHk9LnByZWQsY29sb3I9bW9kZWxfbmFtZSkpICsKICBnZW9tX3BvaW50KCkgKwogIGdlb21fc21vb3RoKG1ldGhvZCA9ICJsbSIpICsKICBsYWJzKHggPSAiT2JzZXJ2ZWQgcHJpY2UiLAogICAgICAgeSA9ICJQcmVkaWN0ZWQgcHJpY2UiLAogICAgICAgdGl0bGUgPSAiUHJlZGljdGlvbnMgdnMgb2JzZXJ2ZWQgdmFsdWVzIGZvciAzIG1vZGVsIHR5cGVzXG5BIHNpbXBsZSBsaW5lYXIgcmVncmVzc2lvbiBpcyBvdmVybGFpZCIpICsKICBjb29yZF9lcXVhbCgpCmBgYAoKCiMjIE1vZGVsIHBlcmZvcm1hbmNlIGFzc2Vzc21lbnQKClVzaW5nIG91ciBwcmVkaWN0aW9ucyBmcm9tIHRoZSBgcGFyc25pcGAgcGFja2FnZSwgd2UgY2FuIHVzZSB0aGUgYHlhcmRzdGlja2AgcGFja2FnZSB0byBnZW5lcmF0ZSBtb2RlbCBwZXJmb3JtYW5jZSBtZXRyaWNzLiBUaGlzIGNhbiBiZSBkb25lIHVzaW5nIHRoZSBgbWV0cmljc2AgZnVuY3Rpb24sIHdoaWNoIGdlbmVyYXRlcyBhIGRlZmF1bHQgbWV0cmljIHNldCAoZm9yIGEgcmVncmVzc2lvbiBtb2RlbCwgdGhlc2UgYXJlIHJvb3QgbWVhbiBzcXVhcmVkIGVycm9yIG9yIGByc21lYCwgci1zcXVhcmVkIG9yIGByc3FgLCBhbmQgbWVhbiBhYnNvbHV0ZSBlcnJvciBvciBgbWFlYDsgZm9yIGEgY2xhc3NpZmljYXRpb24gbW9kZWwsIHRoZXNlIGFyZSBgYWNjdXJhY3lgIGFuZCBLYXBwYSBvciBga2FwYCkuIFlvdSBjYW4gYWxzbyBkZWZpbmUgYSBjdXN0b20gc2V0IG9mIG1ldHJpY3MgdXNpbmcgYG1ldHJpY19zZXRgLCBhbmQgdGhlcmUgYXJlIGFsc28gaW5kaXZpZHVhbCBmdW5jdGlvbnMgZm9yIGFsbCBtZXRyaWMgdHlwZXMuCgpgYGB7cn0KaG91c2luZ19yYW5nZXJfcHJlZGljdCAlPiUKICAjIEhlcmUgd2UgdXNlIHRoZSBtZXRyaWMgZnVuY3Rpb25zIGFuZCBtdXN0IGRlZmluZSB0aGUgdHJ1dGggdmFsdWUgYW5kIHRoZSBlc3RpbWF0ZWQgcHJlZGljdGlvbgogIG1ldHJpY3ModHJ1dGggPSBwcmljZSwgZXN0aW1hdGUgPSAucHJlZCkKYGBgCgpXaGVuIHRoZSBwcmVkaWN0aW9ucyBhcmUgaW4gYSBkYXRhZnJhbWUsIHdlIGNhbiBncm91cCBieSBtb2RlbCB0eXBlIGFuZCBjYWxjdWxhdCBtZXRyaWNzIGJ5IGdyb3VwCgpgYGB7cn0KaG91c2luZ19hbGxfcHJlZGljdCU+JQogIGdyb3VwX2J5KG1vZGVsX25hbWUpICU+JQogIG1ldHJpY3ModHJ1dGggPSBwcmljZSwgZXN0aW1hdGUgPSAucHJlZCkKCmhvdXNpbmdfYWxsX3ByZWRpY3QlPiUKICBncm91cF9ieShtb2RlbF9uYW1lKSAlPiUKICBtZXRyaWNzKHRydXRoID0gcHJpY2UsIGVzdGltYXRlID0gLnByZWQpJT4lCiAgZ2dwbG90KGFlcyh4ID0gbW9kZWxfbmFtZSwgeSA9IC5lc3RpbWF0ZSkpICsKICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIpICsKICBmYWNldF93cmFwKC5+Lm1ldHJpYyxzY2FsZXM9ImZyZWUiKSArCiAgbGFicyh4ID0gIk1vZGVsIG5hbWUiLAogICAgICAgeSA9ICJNb2RlbCBwZXJmb3JtYW5jZSBtZXRyaWMgZXN0aW1hdGUiLAogICAgICAgdGl0bGUgPSAiTW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljcyBmb3IgMyBtb2RlbCB0eXBlcyIpCmBgYAoKV2UgY2FuIGFsc28gZG8gd2hhdCB3ZSBqdXN0IGRpZCBpbiBhIG11Y2ggbW9yZSB0aWR5IGZhc2hpb24sIHdoaWxlIGFsc28ga2VlcGluZyB0aGUgbW9kZWwgc3BlY2lmaWNhdGlvbnMsIG1vZGVsIGZpdHMsIG1vZGVsIHByZWRpY3Rpb25zLCBhbmQgbW9kZWwgbWV0cmljcyBhbGwgaW4gYSBzaW5nbGUgZGF0YWZyYW1lLiBUaGlzIGVuc3VyZXMgdGhhdCB0aGluZ3Mgc3RheSB0b2dldGhlciwgYW5kIG1ha2VzIGl0IHZlcnkgZWFzeSB0byBleHRyYWN0IHN1bW1hcnkgc3RhdGlzdGljcyBvciBwbG90cy4gYHB1cnJyOjptYXBgIGFuZCBsaXN0IGNvbHVtbnMgbWFrZXMgdGhpcyBhbGwgcG9zc2libGUuIFdlIGNvdWxkIGFwcGx5IHRoaXMgc2FtZSBhcHByb2FjaCB0byBidWlsZCBhbmQgdGVzdCBtYW55IG1vZGVscyBmb3IgY3Jvc3MtdmFsaWRhdGlvbiwgZm9yIGh5cGVycGFyYW1ldGVyIHR1bmluZywgZXRjLgoKYGBge3J9CiMgRGVmaW5lIGEgdGliYmxlIHVzaW5nIG1vZGVsIG5hbWVzIGFuZCB0aGVpciBhc3NvY2lhdGVkIHNwZWNpZmljYXRpb25zCmFsbF9tb2RlbHMgPC0gCiAgdGliYmxlKG1vZGVsX25hbWUgPSAibG0iLAogICAgICAgICBtb2RlbCA9IGxpc3QoaG91c2luZ19tb2RlbF9sbSkpICU+JQogIGFkZF9yb3cobW9kZWxfbmFtZSA9ICJyYW5nZXIiLAogICAgICAgICAgbW9kZWwgPSBsaXN0KGhvdXNpbmdfbW9kZWxfcmFuZ2VyKSkgJT4lCiAgYWRkX3Jvdyhtb2RlbF9uYW1lID0gInJhbmRvbUZvcmVzdCIsCiAgICAgICAgICBtb2RlbCA9IGxpc3QoaG91c2luZ19tb2RlbF9yYW5kb21Gb3Jlc3QpKQoKYWxsX21vZGVscwoKYWxsX21vZGVsX3Jlc3VsdHMgPC0gYWxsX21vZGVscyAlPiUKICAjIEFkZCBhIGNvbHVtbiBmb3IgbW9kZWwgZml0cwogIG11dGF0ZShtb2RlbF9maXQgPSBwdXJycjo6bWFwKG1vZGVsLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH5maXQoLngsIHByaWNlIH4gLiwgZGF0YSA9IGhvdXNpbmdfdHJhaW5pbmdfanVpY2VkKSksCiAgICAgICAgICMgQWRkIGEgY29sdW1uIGZvciBwcmVkaWN0aW9ucwogICAgICAgICBtb2RlbF9wcmVkaWN0aW9ucyA9IHB1cnJyOjptYXAobW9kZWxfZml0LAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfi54ICU+JSAKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJlZGljdChob3VzaW5nX3Rlc3RpbmdfYmFrZWQpICU+JQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBiaW5kX2NvbHMoaG91c2luZ190ZXN0aW5nX2Jha2VkKSksCiAgICAgICAgICMgQWRkIGEgY29sdW1uIGZvciBtb2RlbCBtZXRyaWNzCiAgICAgICAgIG1vZGVsX21ldHJpY3MgPSBwdXJycjo6bWFwKG1vZGVsX3ByZWRpY3Rpb25zLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+bWV0cmljcygueCwgdHJ1dGggPSBwcmljZSwgZXN0aW1hdGUgPSAucHJlZCkpKQoKYWxsX21vZGVsX3Jlc3VsdHMKCiMgVGhpcyBwbG90IGlzIHRoZSBzYW1lIGFzIHRoZSBvbmUgd2UgbWFkZSBhYm92ZQphbGxfbW9kZWxfcmVzdWx0cyAlPiUKICB1bm5lc3QobW9kZWxfbWV0cmljcykgJT4lCiAgZ2dwbG90KGFlcyh4ID0gbW9kZWxfbmFtZSwgeSA9IC5lc3RpbWF0ZSkpICsKICBnZW9tX2JhcihzdGF0PSJpZGVudGl0eSIpICsKICBmYWNldF93cmFwKC5+Lm1ldHJpYyxzY2FsZXM9ImZyZWUiKSArCiAgbGFicyh4ID0gIk1vZGVsIG5hbWUiLAogICAgICAgeSA9ICJNb2RlbCBwZXJmb3JtYW5jZSBtZXRyaWMgZXN0aW1hdGUiLAogICAgICAgdGl0bGUgPSAiTW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljcyBmb3IgMyBtb2RlbCB0eXBlcyIpCmBgYAoKIyBDbGFzc2lmaWNhdGlvbiBleGFtcGxlIHVzaW5nIGBHZXJtYW5DcmVkaXRgCgpMZXQncyBhbHNvIGdvIHRocm91Z2ggYSBjbGFzc2lmaWNhdGlvbiBleGFtcGxlIHVzaW5nIHRoZSBgR2VybWFuQ3JlZGl0YCBkYXRhc2V0IGZyb20gYGNhcmV0YC4gTm93IHdlIHdpbGwgdHJ5IHRvIHByZWRpY3QgY3JlZGl0IHJhdGluZyAoZ29vZCBvciBiYWQpIHVzaW5nIGEgbnVtYmVyIG9mIHByZWRpY3RvcnMuCgojIyBMb2FkIHBhY2thZ2VzCmBgYHtyfQpkYXRhKEdlcm1hbkNyZWRpdCkKCkdlcm1hbkNyZWRpdApgYGAKCiMjIERhdGEgcHJlLXByb2Nlc3NpbmcgCmBgYCB7cn0KIyBTcGxpdCB0aGUgY3JlZGl0IGRhdGFzZXQsIHVzaW5nIDc1JSBvZiB0aGUgZGF0YSBmb3IgdHJhaW5pbmcgYW5kIDI1JSBmb3IgdGVzdGluZywgc3RyYXRpZmllZCBieSBjcmVkaXQgY2xhc3MKIyBUaGlzIG1haW50YWlucyB0aGUgcmF0aW8gb2YgR29vZCBhbmQgQmFkIGNyZWRpdCBjbGFzc2VzIGluIGJvdGggdGhlIHRyYWluaW5nIGFuZCB0ZXN0aW5nIGRhdGFzZXRzCmNyZWRpdF9zcGxpdCA8LSBHZXJtYW5DcmVkaXQgJT4lCiAgYXNfdGliYmxlKCkgJT4lCiAgIyBDb252ZXJ0IG1vc3QgY29sdW1ucyB0byBmYWN0b3JzIHNpbmNlIHRoZXkgYXJlIGJpbmFyaWVzCiAgbXV0YXRlX2F0KHZhcnMoLUR1cmF0aW9uLC1BbW91bnQsLUluc3RhbGxtZW50UmF0ZVBlcmNlbnRhZ2UsLVJlc2lkZW5jZUR1cmF0aW9uLC1BZ2UsLU51bWJlckV4aXN0aW5nQ3JlZGl0cywtTnVtYmVyUGVvcGxlTWFpbnRlbmFuY2UpLAogICAgICAgICAgICBhcy5mYWN0b3IpICU+JQogIGluaXRpYWxfc3BsaXQocHJvcCA9IDAuNzUsIHN0cmF0YSA9ICJDbGFzcyIpCgojIFRoZSB0cmFpbmluZyBmdW5jdGlvbiBjYW4gYmUgdXNlZCB0byBleHRyYWN0IHRoZSB0cmFpbmluZyBkYXRhIGZyb20gdGhlIHJzcGxpdCBvYmplY3QKY3JlZGl0X3RyYWluaW5nIDwtIGNyZWRpdF9zcGxpdCAlPiUKICB0cmFpbmluZygpCgojIFRoZSB0ZXN0aW5nIGZ1bmN0aW9uIGNhbiBiZSB1c2VkIHRvIGV4dHJhY3QgdGhlIHRlc3RpbmcgZGF0YSBmcm9tIHRoZSByc3BsaXQgb2JqZWN0CmNyZWRpdF90ZXN0aW5nIDwtIGNyZWRpdF9zcGxpdCAlPiUKICB0ZXN0aW5nKCkKCmNyZWRpdF9yZWNpcGUgPC0gY3JlZGl0X3RyYWluaW5nICU+JQogIHJlY2lwZShDbGFzcyB+LikgJT4lCiAgIyBSZW1vdmUgYWxsIG5lYXItemVybyB2YXJpYW5jZSBwcmVkaWN0b3JzLCBzdWNoIGFzIGZhY3RvcnMgd2l0aCBvbmx5IG9uZSBsZXZlbAogIHN0ZXBfbnp2KGFsbF9wcmVkaWN0b3JzKCkpICU+JQogICMgc3RlcF9jb3JyIHJlbW92ZXMgaGlnaGx5IGNvcnJlbGF0ZWQgdmFyaWFibGVzCiAgc3RlcF9jb3JyKGFsbF9udW1lcmljKCkpICU+JQogICMgc3RlcF9jZW50ZXIgbm9ybWFsaXplcyBkYXRhIHRvIGhhdmUgYSBtZWFuIG9mIDAKICBzdGVwX2NlbnRlcihhbGxfbnVtZXJpYygpKSAlPiUKICAjIHN0ZXBfc2NhbGUgbm9ybWFsaXplcyBkYXRhIHRvIGhhdmUgYSBzdGFuZGFyZCBkZXZpYXRpb24gb2YgMAogIHN0ZXBfc2NhbGUoYWxsX251bWVyaWMoKSklPiUKICAjIE1ha2UgYWxsIGZhY3RvcnMgZHVtbXkgY29sdW1ucwogIHN0ZXBfZHVtbXkoYWxsX25vbWluYWwoKSwtYWxsX291dGNvbWVzKCkpIAoKY3JlZGl0X3JlY2lwZV9wcmVwcGVkIDwtIGNyZWRpdF9yZWNpcGUgJT4lCiAgIyBwcmVwIHRyYWlucyB0aGUgcmVjaXBlIHVzaW5nIHRoZSB0cmFpbmluZyBkYXRhc2V0CiAgcHJlcCgpCgojIFVzZSB1c2UgdGhlIGp1aWNlIGZ1bmN0aW9uIHRvIGFwcGx5IHRoZSBwcmVwcGVkIHJlY2lwZSB0byB0aGUgdHJhaW5pbmcgZGF0YXNldApjcmVkaXRfdHJhaW5pbmdfanVpY2VkIDwtIGp1aWNlKGNyZWRpdF9yZWNpcGVfcHJlcHBlZCkKCiMgV2UgdXNlIHRoZSBiYWtlIGZ1bmN0aW9uIHRvIGFwcGx5IHRoZSBwcmVwcGVkIHJlY2lwZSB0byB0aGUgdGVzdGluZyBkYXRhc2V0CmNyZWRpdF90ZXN0aW5nX2Jha2VkIDwtIGNyZWRpdF9yZWNpcGVfcHJlcHBlZCAlPiUKICBiYWtlKGNyZWRpdF90ZXN0aW5nKSAKYGBgCgojIyBNb2RlbCB0cmFpbmluZyAgCgpgYGB7cn0KIyBEZWZpbmUgYW5kIGZpdCBhIGxvZ2lzdGljIHJlZ3Jlc3Npb24gbW9kZWwKY3JlZGl0X21vZGVsX2xyIDwtIGxvZ2lzdGljX3JlZygpICU+JQogIHNldF9lbmdpbmUoImdsbSIpIAoKIyBEZWZpbmUgYW5kIGZpdCBhIHJhbmRvbSBmb3Jlc3QgcmVncmVzc2lvbiBtb2RlbCB1c2luZyB0aGUgcmFuZG9tRm9yZXN0IGVuZ2luZS9wYWNrYWdlCmNyZWRpdF9tb2RlbF9yYW5kb21Gb3Jlc3QgPC0gIHJhbmRfZm9yZXN0KHRyZWVzID0gMTAwLCBtb2RlID0gImNsYXNzaWZpY2F0aW9uIikgJT4lCiAgc2V0X2VuZ2luZSgicmFuZG9tRm9yZXN0IikgCgojIERlZmluZSBhbmQgZml0IGEgcmFuZG9tIGZvcmVzdCByZWdyZXNzaW9uIG1vZGVsIHVzaW5nIHRoZSByYW5nZXIgZW5naW5lL3BhY2thZ2UKY3JlZGl0X21vZGVsX3JhbmdlciA8LSByYW5kX2ZvcmVzdCh0cmVlcyA9IDEwMCwgbW9kZSA9ICJjbGFzc2lmaWNhdGlvbiIpICU+JQogIHNldF9lbmdpbmUoInJhbmdlciIpIAoKCiMgRGVmaW5lIGEgdGliYmxlIHVzaW5nIG1vZGVsIG5hbWVzIGFuZCB0aGVpciBhc3NvY2lhdGVkIHNwZWNpZmljYXRpb25zCmFsbF9tb2RlbHNfY3JlZGl0IDwtIAogIHRpYmJsZShtb2RlbF9uYW1lID0gInJhbmdlciIsCiAgICAgICAgICBtb2RlbCA9IGxpc3QoY3JlZGl0X21vZGVsX3JhbmdlcikpICU+JQogIGFkZF9yb3cobW9kZWxfbmFtZSA9ICJyYW5kb21Gb3Jlc3QiLAogICAgICAgICAgbW9kZWwgPSBsaXN0KGNyZWRpdF9tb2RlbF9yYW5kb21Gb3Jlc3QpKQoKYWxsX21vZGVsX3Jlc3VsdHNfY3JlZGl0IDwtIGFsbF9tb2RlbHNfY3JlZGl0ICU+JQogICMgQWRkIGEgY29sdW1uIGZvciBtb2RlbCBmaXRzCiAgbXV0YXRlKG1vZGVsX2ZpdCA9IHB1cnJyOjptYXAobW9kZWwsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgfmZpdCgueCwgQ2xhc3MgfiAuLCBkYXRhID0gY3JlZGl0X3RyYWluaW5nX2p1aWNlZCkpLAogICAgICAgICAjIEFkZCBhIGNvbHVtbiBmb3IgY2xhc3MgcHJlZGljdGlvbnMKICAgICAgICAgbW9kZWxfcHJlZGljdGlvbnNfY2xhc3MgPSBwdXJycjo6bWFwKG1vZGVsX2ZpdCwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH4ueCAlPiUgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIHByZWRpY3QoY3JlZGl0X3Rlc3RpbmdfYmFrZWQsIHR5cGUgPSAiY2xhc3MiKSAlPiUKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgYmluZF9jb2xzKGNyZWRpdF90ZXN0aW5nX2Jha2VkKSksCiAgICAgICAgICMgQWRkIGEgY29sdW1uIGZvciBtb2RlbCBtZXRyaWNzIGZyb20gY2xhc3MgcHJlZGljdGlvbnMKICAgICAgICAgbW9kZWxfbWV0cmljcyA9IHB1cnJyOjptYXAobW9kZWxfcHJlZGljdGlvbnNfY2xhc3MsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIH5tZXRyaWNzKC54LCB0cnV0aCA9IENsYXNzLCBlc3RpbWF0ZSA9IC5wcmVkX2NsYXNzKSksCiAgICAgICAgICMgQWRkIGEgY29sdW1uIGZvciBwcm9iYWJpbGl0eSBwcmVkaWN0aW9ucwogICAgICAgICBtb2RlbF9wcmVkaWN0aW9uc19wcm9iID0gcHVycnI6Om1hcChtb2RlbF9maXQsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+LnggJT4lIAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBwcmVkaWN0KGNyZWRpdF90ZXN0aW5nX2Jha2VkLCB0eXBlPSJwcm9iIikgJT4lCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGJpbmRfY29scyhjcmVkaXRfdGVzdGluZ19iYWtlZCkpLAogICAgICAgICAjIEFkZCBST0MgY3VydmVzIGZyb20gcHJvYmFiaWxpdHkgcHJlZGljdGlvbnMKICAgICAgICAgcm9jX2N1cnZlcyA9IHB1cnJyOjptYXAobW9kZWxfcHJlZGljdGlvbnNfcHJvYiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB+cm9jX2N1cnZlKC54LCBDbGFzcywgLnByZWRfR29vZCkpKQpgYGAKCiMjIE1vZGVsIHBlcmZvcm1hbmNlIGFzc2Vzc21lbnQKCmBgYHtyfQojIExldCdzIHBsb3QgdGhlIHBlcmZvcm1hbmNlIG1ldHJpY3MgYnkgbW9kZWwgdHlwZQphbGxfbW9kZWxfcmVzdWx0c19jcmVkaXQgJT4lCiAgdW5uZXN0KG1vZGVsX21ldHJpY3MpICU+JQogIGdncGxvdChhZXMoeCA9IG1vZGVsX25hbWUsIHkgPSAuZXN0aW1hdGUpKSArCiAgZ2VvbV9iYXIoc3RhdD0iaWRlbnRpdHkiKSArCiAgZmFjZXRfd3JhcCgufi5tZXRyaWMsc2NhbGVzPSJmcmVlIikgKwogIGxhYnMoeCA9ICJNb2RlbCBuYW1lIiwKICAgICAgIHkgPSAiTW9kZWwgcGVyZm9ybWFuY2UgbWV0cmljIGVzdGltYXRlIiwKICAgICAgIHRpdGxlID0gIk1vZGVsIHBlcmZvcm1hbmNlIG1ldHJpY3MgZm9yIDIgbW9kZWwgdHlwZXMiKQoKIyBMZXQncyBhbHNvIHBsb3QgUk9DIGN1cnZlcyBieSBtb2RlbCB0eXBlCmFsbF9tb2RlbF9yZXN1bHRzX2NyZWRpdCAlPiUKICB1bm5lc3Qocm9jX2N1cnZlcykgJT4lCiAgZ2dwbG90KGFlcyh4ID0gMS1zcGVjaWZpY2l0eSx5PXNlbnNpdGl2aXR5LGNvbG9yPW1vZGVsX25hbWUpKSArCiAgZ2VvbV9saW5lKCkgKwogIGdlb21fYWJsaW5lKHNsb3BlPTEpICsKICBsYWJzKHRpdGxlID0gIlJlY2VpdmVyIE9wZXJhdGluZyBDaGFyYWN0ZXJpc3RpYyAoUk9DKSBjdXJ2ZXMgZm9yIDIgbW9kZWwgdHlwZXMiLAogICAgeCA9ICJGYWxzZSBwb3NpdGl2ZSByYXRlXG5bMSAtIHNwZWNpZmljaXR5ID0gMSAtIFROLyhUTiArIEZQKV0iLAogICAgeSA9ICJUcnVlIHBvc2l0aXZlIHJhdGVcbltyZWNhbGwgPSBzZW5zaXRpdml0eSA9IFRQIC8gKFRQICsgRk4pXSIpCgpgYGAKCg==